Recent years have witnessed the rapid progress of image captioning. However, the demands for large memory storage and heavy computational burden prevent these captioning models from being deployed on mobile devices. The main obstacles lie in the heavyweight visual feature extractors (i.e., object detectors) and complicated cross-modal fusion networks. To this end, we propose LightCap, a lightweight image captioner for resource-limited devices. The core design is built on the recent CLIP model for efficient image captioning. To be specific, on the one hand, we leverage the CLIP model to extract the compact grid features without relying on the time-consuming object detectors. On the other hand, we transfer the image-text retrieval design of CLIP to image captioning scenarios by devising a novel visual concept extractor and a cross-modal modulator. We further optimize the cross-modal fusion model and parallel prediction heads via sequential and ensemble distillations. With the carefully designed architecture, our model merely contains 40M parameters, saving the model size by more than 75% and the FLOPs by more than 98% in comparison with the current state-of-the-art methods. In spite of the low capacity, our model still exhibits state-of-the-art performance on prevalent datasets, e.g., 136.6 CIDEr on COCO Karpathy test split. Testing on the smartphone with only a single CPU, the proposed LightCap exhibits a fast inference speed of 188ms per image, which is ready for practical applications.

Despite the remarkable progress of image captioning, existing captioners typically lack the controllable capability to generate desired image captions, e.g., describing the image in a rough or detailed manner, in a factual or emotional view, etc. In this paper, we show that a unified model is qualified to perform well in diverse domains and freely switch among multiple styles. Such a controllable capability is achieved by embedding the prompt learning into the image captioning framework. To be specific, we design a set of prompts to fine-tune the pre-trained image captioner. These prompts allow the model to absorb stylized data from different domains for joint training, without performance degradation in each domain. Furthermore, we optimize the prompts with learnable vectors in the continuous word embedding space, avoiding the heuristic prompt engineering and meanwhile exhibiting superior performance. In the inference stage, our model is able to generate desired stylized captions by choosing the corresponding prompts. Extensive experiments verify the controllable capability of the proposed method. Notably, we achieve outstanding performance on two diverse image captioning benchmarks including COCO Karpathy split and TextCaps using a unified model.

Strong lensing in galaxy clusters probes properties of dense cores of dark matter halos in mass, studies the distant universe at flux levels and spatial resolutions otherwise unavailable, and constrains cosmological models independently. The next-generation large scale sky imaging surveys are expected to discover thousands of cluster-scale strong lenses, which would lead to unprecedented opportunities for applying cluster-scale strong lenses to solve astrophysical and cosmological problems. However, the large dataset challenges astronomers to identify and extract strong lensing signals, particularly strongly lensed arcs, because of their complexity and variety. Hence, we propose a framework to detect cluster-scale strongly lensed arcs, which contains a transformer-based detection algorithm and an image simulation algorithm. We embed prior information of strongly lensed arcs at cluster-scale into the training data through simulation and then train the detection algorithm with simulated images. We use the trained transformer to detect strongly lensed arcs from simulated and real data. Results show that our approach could achieve 99.63 % accuracy rate, 90.32 % recall rate, 85.37 % precision rate and 0.23 % false positive rate in detection of strongly lensed arcs from simulated images and could detect almost all strongly lensed arcs in real observation images. Besides, with an interpretation method, we have shown that our method could identify important information embedded in simulated data. Next step, to test the reliability and usability of our approach, we will apply it to available observations (e.g., DESI Legacy Imaging Surveys) and simulated data of upcoming large-scale sky surveys, such as the Euclid and the CSST.

Video super-resolution is one of the most popular tasks on mobile devices, being widely used for an automatic improvement of low-bitrate and low-resolution video streams. While numerous solutions have been proposed for this problem, they are usually quite computationally demanding, demonstrating low FPS rates and power efficiency on mobile devices. In this Mobile AI challenge, we address this problem and propose the participants to design an end-to-end real-time video super-resolution solution for mobile NPUs optimized for low energy consumption. The participants were provided with the REDS training dataset containing video sequences for a 4X video upscaling task. The runtime and power efficiency of all models was evaluated on the powerful MediaTek Dimensity 9000 platform with a dedicated AI processing unit capable of accelerating floating-point and quantized neural networks. All proposed solutions are fully compatible with the above NPU, demonstrating an up to 500 FPS rate and 0.2 [Watt / 30 FPS] power consumption. A detailed description of all models developed in the challenge is provided in this paper.

二进制代码相似性检测（BCSD）方法测量了两个二进制可执行代码的相似性。最近，基于学习的BCSD方法取得了巨大的成功，在检测准确性和效率方面表现优于传统的BCSD。但是，现有的研究在基于学习的BCSD方法的对抗脆弱性上相当稀疏，这会导致与安全相关的应用程序危害。为了评估对抗性的鲁棒性，本文设计了一种高效且黑色的对抗代码生成算法，即FuncFooler。 FuncFooler限制了对抗代码1）保持程序的控制流程图（CFG）和2）保持相同的语义含义。具体而言，funcfooler连续1）在恶意代码中确定脆弱的候选人，2）从良性代码中选择和插入对抗性指令，以及3）纠正对抗代码的语义副作用以满足约束。从经验上讲，我们的FuncFooler可以成功攻击包括Safe，ASM2VEC和JTRAN在内的三种基于学习的BCSD模型，它们质疑是否需要基于学习的BCSD。

我们与中国的援助卫生委员会合作，我们提出了一个预测系统，以根据免疫后不良事件的数据来预测患有不良反应的儿童的住院。我们从数据中提取了多个功能，并选择“住院或不选择”作为分类目标。由于数据是不平衡的，因此我们使用了各种班级不平衡学习方法来培训并改善了Rusboost算法。实验结果表明，在这些算法中，ROC曲线在ROC曲线下的最高面积是最高的。此外，我们将这些平衡的学习方法与一些常见的机器学习算法进行了比较。我们将改进的Rusboost与动态Web资源开发技术结合在一起，以构建一个评估系统，并为相关医生提供信息输入和疫苗接种响应预测能力。

自我监督的学习方法，如对比学习，在自然语言处理中非常重视。它使用对培训数据增强对具有良好表示能力的编码器构建分类任务。然而，在对比学习的学习成对的构建在NLP任务中更难。以前的作品生成单词级更改以形成对，但小变换可能会导致句子含义的显着变化作为自然语言的离散和稀疏性质。在本文中，对对抗的训练在NLP的嵌入空间中产生了挑战性和更难的学习对抗性示例作为学习对。使用对比学学习提高了对抗性培训的泛化能力，因为对比损失可以使样品分布均匀。同时，对抗性培训也提高了对比学习的稳健性。提出了两种小说框架，监督对比对抗学习（SCAS）和无监督的SCAS（USCAL），通过利用对比学习的对抗性培训来产生学习成对。利用基于标签的监督任务丢失，以产生对抗性示例，而无监督的任务会带来对比损失。为了验证所提出的框架的有效性，我们将其雇用到基于变换器的模型，用于自然语言理解，句子语义文本相似性和对抗学习任务。胶水基准任务的实验结果表明，我们的微调监督方法优于BERT $ _ {基础} $超过1.75 \％。我们还评估我们对语义文本相似性（STS）任务的无监督方法，并且我们的方法获得77.29 \％with bert $ _ {base} $。我们方法的稳健性在NLI任务的多个对抗性数据集下进行最先进的结果。

我们提出了用于将Swin变压器缩放到3亿参数的技术，并使其能够使用高达1,536美元的图像培训1,536美元。通过缩放容量和分辨率，Swin变压器在四个代表视觉基准上设置新记录：84.0％的Top-1在Imagenet-V2图像分类准确度，63.1 / 54.4盒/掩模地图上的Coco对象检测，59.9 Miou在Ade20K语义细分中，在动力学-400视频动作分类上的86.8％的前1个精度。我们的技术通常适用于缩放视觉模型，这尚未广泛探索为NLP语言模型，部分原因是培训和应用中的困难：1）视觉模型经常面临规模的不稳定问题，2）许多下游愿景任务需要高分辨率图像或窗口，并且目前尚不清楚如何有效地将模型在低分辨率上预先培训到更高分辨率。当图像分辨率高时，GPU存储器消耗也是一个问题。为了解决这些问题，我们提出了几种技术，通过使用Swin Transformer作为案例研究来说明：1）归一化技术和缩放的余弦注意力，提高大视觉模型的稳定性; 2）一种日志间隔的连续位置偏置技术，以有效地将在低分辨率图像和窗口预先训练的模型转移到其更高分辨率的对应物。此外，我们分享了我们的关键实施细节，导致GPU内存消耗的大量节省，从而使得用常规GPU培训大型视觉模型可行。使用这些技术和自我监督的预训练，我们成功培训了强大的3B往返变压器模型，并有效地将其转移到涉及高分辨率图像或窗口的各种视觉任务，实现了各种最先进的准确性基准。

We evaluate whether features extracted from the activation of a deep convolutional network trained in a fully supervised fashion on a large, fixed set of object recognition tasks can be repurposed to novel generic tasks. Our generic tasks may differ significantly from the originally trained tasks and there may be insufficient labeled or unlabeled data to conventionally train or adapt a deep architecture to the new tasks. We investigate and visualize the semantic clustering of deep convolutional features with respect to a variety of such tasks, including scene recognition, domain adaptation, and fine-grained recognition challenges. We compare the efficacy of relying on various network levels to define a fixed feature, and report novel results that significantly outperform the state-of-the-art on several important vision challenges. We are releasing DeCAF, an open-source implementation of these deep convolutional activation features, along with all associated network parameters to enable vision researchers to be able to conduct experimentation with deep representations across a range of visual concept learning paradigms.

In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed implicitly, by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. CMT obtains 73.0% NDS on nuScenes benchmark. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code will be released at https://github.com/junjie18/CMT.